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INTRODUCTION 


Choosing a school for a child can be a rewarding but difficult task for parents.! 
A recent survey reported that parents often feel confused by the information 
available to them as part of the school choice process.” This can be exacer- 
bated by limited time and resources to devote to the process—constraints that 
can particularly affect low-income and otherwise disadvantaged families.’ If 
confusion results in uninformed decision-making then parents might not select 
schools that best fit their children’s needs and the possible benefits of school 
choice could be undermined. 


Understanding how information can influence parents’ school choice behavior 

is important now, more than ever. In many parts of the country, opportunities to 
choose a school are increasing. Nationwide, the number of charter schools, magnet 
schools, and private school voucher programs has risen in recent years, as have 


open enrollment policies that allow parents to choose from among traditional 
public schools.* For example, over the most recent five years for which data are available, the number of magnet schools 
increased by almost 50 percent and the number of charter schools grew by more than a third (see Figure 1). 


‘There is also new demand for making school information available to families. The federal Every Student Succeeds 
Act (ESSA) passed in 2015 requires states and districts to provide public report cards that include information on 
state, district, and school performance and progress in an “understandable” and “user-friendly” manner.’ These report 
cards are likely to build on the school choice directories (online, on paper, or both) that many districts, states, and 
other education organizations already offer to help families navigate their options. 


Figure 1. Number of Charter and Magnet Schools Has Grown Over Time 


Number of Schools 
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Source: National Center for Education Statistics, Common Core of Data, Public Elementary/Secondary School Universe Survey. 
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Currently, how information is provided in school directories or report cards varies in both substance and style.° For 
example, some provide lengthy and detailed profiles for each school, while others opt for more concise and modest 
profiles. Some present information primarily with numbers, while others use graphs or icons such as color-coded 
letter grades. Research in fields outside of education indicates that the way information is presented affects deci- 
sion making.’ But little is currently known about how those results apply to school choice information and whether 
different strategies might help or hinder parents as they go through the choice process. The U.S. Department of 
Education’s guidance on ESSA report cards suggests there is more to be learned about best practices for displaying 
school information and that obtaining feedback from parents is essential.* 


The Department, which also financially supports school choice programs through initiatives such as the Public 
Charter Schools Program and the Magnet Schools Assistance Program, seeks to improve the evidence on how to 
present information about schools to the public. This evidence-based guide explores how presentation decisions can 
affect parents’ understanding, ease of use, and satisfaction with the school information provided, and can affect how 
parents rank their school choices. The evidence that informs the guide comes from a rigorous experiment conducted 
in late 2016 for the Department’s Institute of Education Sciences.’ 


This guide is a resource for those considering how to present information to parents (“information providers”). The 
recommendations in the guide are “evidence-based” because they emerged from a randomized study that is described 
below. The intended audience for the guide includes, for example, education officials responsible for deciding which 
information about schools to release publicly, along with designers responsible for translating that information into 
carefully formatted materials. Some information providers work in public agencies, like school districts or state 
departments of education, but many others work in nonprofit organizations or other private companies. The purpose 
of this guide is to help information providers—whoever and wherever they are—incorporate evidence into the many 
decisions involved in presenting information about schools. Table 1 explains some of the terms used in this guide. 


Table 1. Terms Used in this Guide 


Information display | The full display that parents see when they visit a school choice website. For the study, 
this included, for example, a map, glossary of terms, and 16 individual school profiles. 


School profile The part of an information display containing information about a single school. The school 
profiles appear as rows in a table. 


Measure A specific piece of information about a school. Measures were grouped into four different 
“categories of information” (distance from home, academics, safety, and resources). For 
example, the number of laptops or tablets per 100 students is a measure within the resources 
category of information. 


Factor An aspect of the information displays that the research team varied such that different study 
participants saw different displays. The study examined five factors. For example, the “amount 
of information” factor varied how much information about each school was shown. 


Strategy One way of implementing a particular factor. For example, the “amount of information” factor 
tested three strategies: lower amount, higher amount, and progressive disclosure (where 
parents initially viewed a lower amount but had the option to click a link that revealed more 
information). 


Variation of an Any specific version of an information display used in the study. The study tested 72 variations, 
information display | because there were 72 possible combinations of strategies across the five factors. 


Note: Terms in bold are defined in this table. 
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THE STUDY AND HOW IT WORKED 


For the study, researchers worked with a firm specializing in presenting education data to create an online “information 
display” of individual “school profiles” for a hypothetical school district. Parents who participated in the study were each 
randomly assigned to review one variation of the information display—as if they were considering elementary school 
options for their youngest child—and then answered questions about the display and the schools it described. This type 
of experiment is commonly used for market research and by behavioral and social scientists to examine the effects of 
specific variations (e.g., color of a product, wording of instructions, etc.) in controlled environments. For this study, the 
research team compared responses to the questions for parents who saw different information displays to determine how 
the design of the displays affected key outcomes: parents’ understanding of the school information, their satisfaction 
with the display and its perceived ease of use, and the choices they made when ranking the schools. 


Figure 2 illustrates each step of the study and is followed by a description of how the study worked and what specifically 
was tested. A more detailed discussion of the study and its methods appears in the technical appendix, Section B.1. 


Figure 2. How the Study Worked 


Study team recruits low-income parents 


(A) Parents complete initial survey 
e Answer questions to confirm study eligibility 
e Provide background information 


Study team randonlly assigns parents to 1 of 72 displays of 
school information ("information displays”) 


(B) Parents view school information display including profiles 
for 16 hypothetical schools 


(C) Parents complete outcomes survey 
e Rank top 3 preferred schools for their child 
e Take test to demonstrate “understanding” of display 
e Rate ease of use and satisfaction with display 


Study team compares outcomes across parents viewing different displays 
to see which display strategies are best 
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Recruitment and Participants 


Parents were recruited by a market research firm that maintains a database of potential survey respondents. Partici- 
pants received modest compensation from the firm if they were eligible for and participated in this particular study. 


Those interested in participating began by completing an initial survey to determine if they met eligibility require- 
ments, which included living with at least one school-age child (aged 3 to 18) for whom they helped make education 
related decisions. To participate parents also had to be low-income (annual household income of $40,000 or less) 
because research indicates that these parents face particular challenges related to accessing and utilizing information 
about schools. Also, many of today’s school choice programs specifically target low-income families." 


In total, 3,500 parents from across all 50 states participated in the study and are represented, anonymously, in the 
study’s findings. In addition to household income, the initial survey gathered information about these parents’ educa- 
tional attainment, Internet use, and prior experiences with choosing schools."' This enabled researchers to determine 
if particularly disadvantaged parents, those familiar with online information displays, or those with school choice 
experience tended to respond to school information displays differently from other groups. 


Viewing the Information Displays 


After completing the initial survey, parents viewed an information 
display with profiles of 16 schools from a hypothetical district. The 
schools were designed to differ from one another with respect to 
their distance from the parent’s hypothetical home and the infor- 
mation reported about school academics, safety, and resources (such 
as number of laptops). The research team generated this hypothet- 
ical information to create trade-offs between schools—for example, 
the school located closest to home also had relatively poor academic 
performance, low safety, and few resources compared to other 
schools. Parents were asked to imagine that they were selecting 

an elementary school for their youngest child’ as they examined 
the profiles and used them to compare schools, because the study’s 


information display included only elementary schools. 


An important aspect of this study is that each parent was assigned, at random," to see one of 72 variations of the 
school information display (see next section). However, there were four features common to all information displays 
(as shown in Figure 3). They are: 


1. aglossary that defines key terms such as “proficiency rate” that may be unfamiliar to some parents; 


2. amap with a house symbol representing the parent’s hypothetical home and markers representing the location 
of each school; 


3. 16 school profiles with information about each school’s distance from home, academics, safety, and resources 
which are information categories that parents report caring about'*”; and 


4. a simple, standard data layout that made it possible to scan up or down and compare the 16 different schools 
on each information category. 
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Figure 3. Features Common to all Information Displays 


Title banner with link 
to glossary and option 
to display or hide map 


HIDE MAP 


Map with home symbol 
and pins representing 
locations of 16 schools 


Profile banner with “Sort 


By” dropdown list and 
labels for each category 
of information followed 
by 16 rows containing 
school profiles 


Four categories of 
information: distance, 
academics, safety, and 
resources 


‘The features noted above were adopted because they are consistent with beliefs about best design practices from the 
research literature’? and those recommended by the study’s design firm, a company with experience creating school 
information displays for a variety of state and district customers. (See the appendix for further discussion of why 
information providers might consider these features when developing their own materials.) The goal was to avoid 
any designs that are known to hinder understanding, satisfaction, and ease of use (outcomes discussed in more detail 
below) and instead focus on testing potentially promising strategies against one another. 


Outcomes 


After parents viewed the specific information display that they had been randomly assigned to see, they began the “outcomes 
survey.” (The school information display remained visible, so parents could refer to it as needed.) The study varied what 
parents saw in order to measure the effects of any one strategy on understandability, ease of use, satisfaction, and choices 
made by parents. These outcomes are important to consider when evaluating school information displays, according to 
both ESSA report card guidance and prior school choice research.” The outcomes were defined and measured as follows: 


wie Understanding (comprehension of the information provided). To assess parents’ understanding 
= @ of the school profiles, the survey asked six factual (right-or-wrong) questions that required 
© parents to refer to information in the school profiles. Parents could consult the profiles 
= while answering the questions. The researchers assigned points for correct answers and 
calculated the percentage of total possible points that each parent earned. A higher 
percentage reflects greater understanding. 


Example: The survey asked parents which schools, from a list, are within one mile 
of home and have at least 50 laptops or tablets per 100 students. 
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parents’ attitudes about ease of use, six survey items asked parents how much they agreed 
or disagreed with statements about the ease or difficulty of using the profiles. Responses 
were combined into an ease-of-use index that captured the percentage of statements with 
which parents agreed or strongly agreed. A higher percentage reflects greater ease of use. 


t+ Ease of use (self-reported ability to use the display to learn about schools). To assess 


Example: The survey asked parents whether they would need someone to help them in 
order to use the school profiles effectively. 


Satisfaction (extent to which parents reported liking the display). To assess parents’ 

_ Satisfaction, four survey items asked parents how much they agreed or disagreed with 
Ll statements about their satisfaction with the profiles. Responses were combined into a 
satisfaction index that captured the percentage of statements with which 

parents agreed or strongly agreed. A higher percentage reflects greater satisfaction. 


Example: The survey asked parents if they would like to use these information 
displays to choose a school for their own children. 


Choices (decisions based on the information in the display). To measure the effect of 
information on the types of schools chosen, the survey included a school-ranking exercise. 
After reviewing the profiles, parents completed a two-step process to rank their preferred 
schools as if they were choosing a school for their youngest child. First, they selected their 
top three schools from the 16 schools listed. Then, they ranked those three schools from 
most to least preferred. By looking at the characteristics of the schools selected, researchers 
calculated how much weight parents gave to certain categories of school information, such 
as distance from home and academic performance, when making school selections. 


In order to determine how variations of the information display and specific 
display strategies led to different outcomes, the study team compared 
outcomes for parents who saw any given strategy to outcomes for parents who 


were assigned to see a different strategy. Random assignment of parents to 
different information displays ensures that differences in average outcomes 
can be attributed to the differences in information 
display strategies. A more detailed descrip- 

tion of how outcomes were analyzed 

is in the technical appendix, Section B. 
This guide only reports on results 

with a high probability, given the 

data, that one strategy was better 

than the other(s). 
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WHAT THE STUDY TESTED 


Creating a school information display requires an information provider to make many more decisions than any single 
study could examine. The research team, with guidance from a diverse group of practitioners and scholars, chose to 
investigate a set of potentially consequential decisions where little is known about how design choices relate to the 
study’s outcomes of interest. Specifically, the research team varied five aspects of the information displays (called 
“factors”), testing two or three versions of each (called “strategies”), for a total of 72 variations. Figure 4 shows an 
example of an information display from the study and highlights the factors that were varied. Additional examples 

of all the strategies are presented later in the guide along with more detail concerning why the factors and strategies 
that were tested might affect outcomes of interest. 


Figure 4. Features of the Information Display that Varied in the Study 


Factor 1: Format. Whether data were 
presented as numbers only, numbers 
plus icons, or numbers plus graphs. 


15, .] Wi 9 LL 
Factor 2: District average. Whether 9 : 
display included a row with district 9 e 
wide averages for academics, safety, ; 
and resource measures or not. 5 ; 
@ ~ DISTRICT-WIDE AVERAGE Saeeey GS% suspensions, gy LzptapE Or Tablets 

eT es per 100 Students 

Parent Satisfaction Parent Satisfaction 

with Academics = 6 Q% with Safety 75% 


Factor 3: Data sources. Whether —— a 
display included only information @ @ Lincofs Elementary School Milesfrom ate Sq Suspensions 
provided by the district or included 

both information provided by 
the district and parent ratings of —— 
academics and safety. ls Heer 


42 


75% 


DISTANG ACADEMICS SAFETY 


Factor 4: Amount of information. 
Whether profiles displayed lower 
amount of information, higher 
amount of information, or initially 
displayed lower amount but included 
an option to click a link that revealed 
more information (i.e., progressive 
disclosure, as shown here). 


SCHOpPL INFORMATION 


@ 9 Horace Mann Elementary School 


Factor 5: Default sort order. Whether 
display initially showed schools 


sorted by distance to school from 
home or by academic performance. 


Note: Each parent was assigned at random to view one variation of the information display, each of which represented a unique combination of five 
strategies tested in the study. For example, a parent who saw a display like the one shown above was assigned to see the following strategies: numbers 
plus graphs, district averages, official district-provided information plus parent ratings, progressive disclosure, and a default sort order based on the 
distance from home. This is one of the 72 variations tested in the study. ” 
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ORGANIZATION OF THE GUIDE 


‘The five factors along with the specific information display strategies that were varied for each (see Figure 4) were 
chosen to allow the study to answer six questions. The first five questions, each of which focuses on a single factor, ask 
which strategies make school information displays most understandable, easy to use, and satisfying to parents. The sixth 
question, which looks across all five factors, asks how design decisions might affect the school choices parents make. 


The next section of this guide—“Study Findings and Implications”— addresses each of the six questions, explaining 
why each question was important to answer and what the study found. That section is followed by a discussion— 
“Key Insights From the Study”—of the researchers’ interpretations of the results. The final section discusses some 
reasons why the results and implications should not be viewed as definitive. In particular, they come from a single 
study, conducted online, where parents were asked to consider 16 hypothetical elementary schools. Findings may be 
different in real-world settings with parents deciding where to enroll children in school, especially for parents who 
might have more (or fewer) than 16 schools to choose from or may be choosing middle or high schools. 


Additional details about the study’s sample, methods, and findings appear in the accompanying technical appendix. 
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STUDY FINDINGS AND IMPLICATIONS 


Question 1 


Which FORMAT for displaying data on schools is best— 
numbers only, numbers plus icons, or numbers plus graphs? 


Why and how format was tested 


One of the most important issues information providers have to grapple with is how to present numerical data—such as 
how many students in a school tested as proficient on a state assessment. Not only does ESSA require that report cards 
include a variety of achievement performance and progress indicators (in an understandable and user-friendly way) but 
also parents report that such information, along with information about school safety, is what they care about most.” 


There are multiple options for displaying numerical data, from presenting numbers and percentages on their own to 
adding visual elements such as graphs or charts to creating icons such as letter grades (A-F) or color-coded symbols. 
Research outside of education indicates that graphs and icons can help people organize and interpret information”! 
and indeed there are districts and third party information providers using these approaches.” However, it is not 
actually known if, in the context of school information, alternative formats make numerical data easier to understand, 
are preferred by parents, and are worth the effort required to create them. 


The study tested three format strategies:?*”4 


Numbers Only 
) 


0.1 54% 91% 42 


g Lincoln Elementary School 


12) Ward Elementary School Pe 87% oe : ; et 


Numbers + Icons Letter grade icons 


range from A-F and 


are color- 

Q Lincoln Elementary School 0.1 e coded: A=green, 

. B=yellow, C=orange, 
D/F=red 


.?] Ward Elementary School is 


Numbers + Graphs 
Horizontal bar graphs 


show rates ona scale 


9 
i Profic " Student ith No from 0-100% 
q@ Lincoln Elementary School Miles from ae - Faas. : 
Home 0.1 —————sl us pace ali sh per 100 Students 42 
: Proficiency Students with No 
2 ] Ward Elementary School Miles from Rate 87% Suspensions 89o, Epps or Tablets 
: Home 0.3 sees Se per 100 Students 39 


Note: Examples here use the default distance sort, lower-information strategy, no district averages, and no parent ratings. 


As shown, these strategies were varied for academic and safety information (where parents may seek help interpreting 
information) but not for distance from home and school resources. 
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Question 1 Findings 
Changing the format improved understanding by up to 3 percentage points and improved satisfaction by up 
to 1 percentage point. 
e Numbers only was the most understandable format (i.e., parents were most likely 
to answer factual questions correctly). 
e All three formats were similarly easy to use (i.e., parents reported all were similarly 
easy to use). 


e Numbers plus graphs was the most satisfying format (i.e., parents reported liking it the most). 


Format (Mean 0-100 scale) 


ale 
Siratey -@: Understanding oi Ease of Use Tae Satisfaction 
Numbers Only 75.0" 86.0 90.0 
Numbers + Icons HiZe5 86.0 89.5 
Numbers + Graphs 72.0 86.0 90.5“ 


Note: The caret (4) and bold blue text indicate strategies that have at least a 70 percent chance of producing the greatest understanding, 
ease of use, or satisfaction compared to the other strategies tested. N= 3,500. The understanding scale represents the percentage 
correct on a comprehension test. Other scales represent the average percentage of ease of use and satisfaction statements with which 
parents agreed or agreed strongly (after reverse-coding any negative statements). 


Implications 


Evidence suggests there may be a trade-off. If making the information display understandable is most important, 


then a numbers-only format may be best. If making it satisfying is most important, then adding graphs to accompany 


numbers may be the better option. 
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Question 2 


Is it useful to show parents DISTRICT AVERAGES on key mea- 
sures of school performance for comparison purposes? 


Why and how inclusion of district averages was tested 


Research in fields outside of education suggests that comparative information (e.g., providing the average cost of 
an auto insurance policy alongside the sticker price for a given insurance plan) can make decision tasks easier for 
choosers and studies have documented that offering reference points (the average cost in the example above) does 
shape consumer choices.** ESSA encourages this type of comparative information in state and district report cards, 
for example by requiring districts to provide information on how academic achievement in a particular school 
compares to the district and the state averages. 


However, in the context of school choice, it is not known if providing parents with such averages helps them make 
sense of data shown for individual schools and whether or not the information is useful as they consider their 
options. On one hand, district averages might give context to information that is presented for an individual school. 
(For example, a parent may see that students score about the same on achievement tests in the school closest to their 
home when compared to the district average.) On the other hand, district averages might simply be seen as clutter, 
adding to the amount of information a parent has to consider, and undermining their ability to understand or use the 
information provided. 


The study tested two strategies: 


Not showing district averages 


| Distance v 


Q@ Lincoln Elementary School 


0.1 54% 91% 42 


t2 ) Ward Elementary School ae ae oe | 39 


District averages shown for 


academics, safety, and resources 
Showing the district average 


Distance v 


DISTRICT-WIDE AVERAGE 


Proficiency Students with Laptops or Tablets 

Rate No Suspensions per 100 Students 
.) Lincoln Elementary School 0.1 54% 91% 42 
a pam 87% oe: 20 


Note: Examples here use the default distance sort, lower-information strategy, numbers only format, and no parent ratings. 


As shown, district averages were presented for academic, safety, and school resources information but not for 
distance from home. 
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Question 2 Findings 


Showing district averages reduced ease of use by 0.5 percentage points and reduced satisfaction by 1 percentage point. 


e Both strategies were similarly understandable. 
e No district average was the easiest to use. 


e No district average was the most satisfying. 


District Average (Mean 0-100 scale) 


7 
Sirateor “O): Understanding oui Ease of Use af Satisfaction 
No district average 73.5 86.5" 90.5" 
District average shown 73:5 86.0 89.5 


Note: The caret (4) and bold blue text indicate strategies that have at least a 70 percent chance of producing the greatest understanding, 
ease of use, or satisfaction compared to the other strategies tested. Understanding scale represents the percentage correct on compre- 
hension test. Other scales represent average percentage of ease of use and satisfaction statements with which parents agreed or agreed 
strongly (after reverse-coding any negative statements). N=3,500. 


Implications 


Evidence suggests that including the district averages would make the information harder to use and less satisfying 
to parents. 


Presenting School Choice Information to Parents: An Evidence-Based Guide 13 


Question 3 


Is it useful to include parents’ opinions of schools as an 
additional SOURCE OF INFORMATION for other parents? 


Why and how source of information was tested 


When choosing schools, parents rely on information from both official sources (such as district reports of students’ 
performance on state tests) and other parents (such as word-of-mouth reports on school quality from their neigh- 
bors).** Some education research suggests that parents value and use opinions offered by other parents about schools 
more so than data that comes from school, district, or state reports.’’ 


However, surveying parents—a main way districts or third party information providers might provide parent 
opinions—can be time-consuming and expensive. Currently it is unclear how information from parent surveys affects 
outcomes such as understanding of and satisfaction with information displays. 


The study tested two strategies: 


District Only: Showing only official data provided by school districts (e.g., test score 
proficiency rates and suspension rates) 


Distance v 


| Q Lincoln Elementary School ioe 7 ne wo . ~ ig 
| Q@ Ward Elementary School 0.3 ; hs oe fa ae 


9 Horace Mann Elementary Schoo! Be . 50% ace 69 si 


District + Parent Ratings: Combining district-provided official data with results from 
a parent satisfaction survey on academics and safety 


Distance v 


Q Lincoln Elementary School 0.1 54% 42 
@ Ward Elementary School 0.3 87% 39 
0.4 50% 69 


.3] Horace Mann Elementary School 


Note: Examples here use the default distance sort, lower-information strategy, 
numbers only format, and no district average. School rankings were the same 
for parent satisfaction measures as measures based on district data for the same 
category of information. 


Added information 
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Question 3 Findings 


Including parents’ opinions reduced understanding by 0.5 percentage points but increased satisfaction by 
2.5 percentage points. 


e Showing district-only data was the most understandable. 
e Both strategies, including or omitting parent ratings, were similarly easy to use. 


e District plus parent ratings was the most satisfying. 


Source of Information (Mean 0-100 scale) 


ate 
Strategy -@: Understanding oi Ease of Use Tae Satisfaction 
District Only 73.5" 86.0 89.0 


District + Parent Ratings 


73.0 | 86.0 91.5% 


Note: The caret (4) and bold blue text indicate strategies that have at least a 70 percent chance of producing the greatest understanding, 
ease of use, or satisfaction compared to the other strategies tested. Understanding scale represents the percentage correct on compre- 
hension test. Other scales represent average percentage of ease of use and satisfaction statements with which parents agreed or agreed 
strongly (after reverse-coding any negative statements). N=3,500. 


Implications 


Evidence suggests there is a trade-off. If making the information display understandable is most important, then 
omitting parent ratings may be best. If making it satisfying is most important, then adding parent ratings to accom- 
pany district provided information may be the better option. 
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Question 4 


What AMOUNT OF INFORMATION about each school should 
be shown in a summary of the choices available to parents? 


Why and how amount of information was tested 


Research in fields outside of education suggests that limiting the amount of information shown can make an infor- 
mation display easier to understand.** On the other hand, some studies show that displays that are too simple can be 
less satisfying because choosers may not feel there is enough information to make good decisions.” Those providing 
school information wrestle with how much information parents want and need to make informed choices. For 
example, ESSA is requiring several pieces of information about each school be included in district provided report 
cards.*° However, a recent review of 14 online school information displays found some offer substantially more infor- 
mation than others when initially showing information for each school profiled.** 


In the context of school choice, there is not much evidence on what is enough information so that parents learn 
about individual schools without presenting so much information that they become overwhelmed or confused. 
Online information displays offer an option to allow parents the ability to start with a lower-amount of information 
and then click on a link to reveal more detailed information. But whether or not such a strategy is actually helpful to 


parents is unknown. 
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The study tested three strategies: 


Lower Amount: one key measure per category of school information (e.g., one measure 
related to school resources) 


: = 
| Distance a 


Q Lincoln Elementary School 


Q Ward Elementary School 


Higher Amount: up to six measures per category (e.g., six measures related to 
school resources) 


Distance v 


Q Lincoln Elementary School 


DISTANCE ACADEMICS SAFETY RESOURCES 
91% 42 


SCHOOL INFORMATION 2001 


www.le.mid.k12.us 
August 29, 2016 Facilities for Extracurricular Activities 


No Yes 
No No 


Added information 


Progressive Disclosure: lower amount of information initially displayed, with 
the parent’s option of clicking to see the higher amount of information for any school. 


Distance v 


0.1 54% 91% 42 
SAFETY RESOURCES 
| Users can click here to see the 45 91% 42 
higher amount of information ’ ' ' 
eis 41 2001 
SCHOOL INFORMATION 90% 
te: www.le.mid.k12.us 56% 4a Att 
August 29, 2016 Nc Facilities for Extracurricular Activities 
ied No Yes 
The lower amount of information ying en ‘ 
is shown by default No No 


6 @ Ward Elementary School we whe sia a 


Note: Examples here use the default distance sort, numbers-only format, no parent ratings, and no district average. 
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Question 4 Findings 


Varying the amount of information improved satisfaction by up to 1.5 percentage points 


e The three strategies were similarly understandable. 
e The three strategies were similarly easy to use. 
e Showing a higher amount of information was the most satisfying. 


Amount of Information (Mean 0-100 scale) 


Se -@: Understanding ogi Ease of Use ae Satisfaction 
Lower Amount VER) 86.5 89.5 
Higher Amount 73.0 86.0 91.0’ 
nicaress ve 73.5 86.0 89.5 


Disclosure 


Note: The caret (4) and bold blue text indicate strategies that have at least a 70 percent chance of producing the greatest understanding, 
ease of use, or satisfaction compared to the other strategies tested. Understanding scale represents the percentage correct on compre- 
hension test. Other scales represent average percentage of ease of use and satisfaction statements with which parents agreed or agreed 
strongly (after reverse-coding any negative statements). N=3,500. 


Implications 


Evidence suggests that the higher information display would be more satisfying to parents. Information providers 
should consider displaying more than a single row of information on each school. 
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Question 5 


Does it matter in what ORDER the school profiles initially 
appear on the information display? 


Why and how default sort order was tested 


When designing school choice information, information providers must select an order in which to present 
schools, whether in a paper or online display. Information displays in current use, such as those in Washington, DC, 
New Orleans, and Detroit, each present the initial list of schools in a different order (distance from home, name, 
and performance, respectively). 


There is little evidence regarding whether initially sorting by distance versus, say, academic performance, affects 
parents’ ability to understand information.*’ Moreover, it seems possible—but has not been demonstrated—that the 
order in which schools are presented may influence how school data are actually used. For example, sort order might 
affect the schools that parents read about and consider, since many parents may not read about every school presented 
in a guide and could be more likely to consider the first schools listed. Researchers working in fields other than 
education have found that ordering people’s options from the highest-quality option first to the lowest-quality option 
last tends to result in people choosing higher-quality options.** This is consistent with the often-observed practice of 
“satisficing,” where people making a choice search only until they find a selection that meets their standards, rather 
than considering and comparing all of the available options.* 


The study tested two defaults for how the list of schools was sorted (though in each case 
parents were able to change the initial sort order): 


Sorting by distance 


The first school is the 


closest to home : g 


an 
© \ 9 


User can select distance, 
academic rating, or alphabetical, 


but distance is default 


Distance v 


SS 
Q Lincoln Elementary School 0.1 54% 91% 42 
Q@ Ward Elementary School 0.3 87% 89% 39 
are 
(continued) 
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Sorting by academic quality 


Vv 


@ \ 9 The first school has the highest 


academic proficiency rate 


User can select distance, 
academic rating, or alphabetical, 


but academic rating is default .< Le mews Ge 


13 


Academic Rating VY 


© OpenStreetMap 


a 
@ Franklin Elementary Schoo 1.9 25% : a1 oe 
t2 ] Peirce Elementary School Zs : se ; ae * ' 


Note: Examples here use the numbers-only format, lower information, no parent ratings, and no district average. 


Question 5 Findings 


Understanding and satisfaction were 1 percentage point higher when the initial sort order was by distance. 


e Sorting by distance was most understandable to parents as a way to list schools by default. 
e The two default sort orders were similarly easy to use. 


e Sorting by distance was the most satisfying way to list schools by default. 


Default Sort Order (Mean 0-100 scale) 


Sirnteor -@: Understanding oi Ease of Use az Satisfaction 
By Distance 74.0’ 86.0 90.5’ 
By Academics 73.0 86.0 89.5 


Note: The caret (4) and bold blue text indicate strategies that have at least a 70 percent chance of producing the greatest understanding, 
ease of use, or satisfaction compared to the other strategies tested. Understanding scale represents the percentage correct on compre- 
hension test. Other scales represent average percentage of ease of use and satisfaction statements with which parents agreed or agreed 
strongly (after reverse-coding any negative statements). N=3,500. 


Implications 


Evidence suggests that sorting by distance may be a better way to make information understandable and satisfying 
to parents. 
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Question 6 


Can information be presented in ways that encourage parents 
to choose an academically higher-performing school? 


Why and how encouragement to select academically higher-performing 
schools was tested 


Unlike Questions 1 through 5, each of which considers the effects of varying one particular factor on understanding, 
ease of use, and satisfaction, Question 6 considers all five factors but focuses on a behavioral outcome—the choices 
that parents make, particularly whether a given strategy makes them more likely to choose schools with higher 
academic performance. 


‘There is reason to believe the five factors this guide examines (format, default sort order, amount of information, and 
so on) might affect choice behavior. For example, it is well known that health insurance choices, food choice, and retail 
shopping are all influenced by the types of factors discussed.*° There is little evidence, however, to guide school choice 
information providers who may want parents to give extra consideration to academically higher performing schools. 


As a reminder, after parents viewed the profiles for the 16 hypothetical schools, they were asked to select and rank 
their top three as if they were choosing a school for their youngest child. Then the research team looked at the types 
of schools that the parents preferred the most. These data were used to test whether different ways of presenting 

the information related to parents’ selecting schools that were higher-performing academically. The term “higher- 
performing” refers only to the proficiency rate at the school, defined for parents as “the percentage of students who 
scored Proficient or better on the 2016 state assessments (average across math and reading for all tested grades) 


” 


D. SCHOOL SELECTION TASK 


Using the school profiles provided in the school finder, please select the three elementary schools from the list below that 


you would most prefer for your youngest child. 


Adams Elementary School 
Now, considering the school profile information, please put the 3 schools you selected in 

DO) Bowen Elementary School order from your first choice (most preferred) on top to your third choice (least preferred) on 
0 Brackett Elementary School aa ecient 


(0 Cabot Elementary School 


Items 
CO Dallin €1 — Rank order your Top 3 schools in 
allin Elementary Schoo aor this box. 


eke Elser Seioot Fiske Elementary School 


O Franklin Elementary School Harrington Elementary 
School 


Harrington Elementary School 


0 Horace Mann Elementary School 


O Lincoln Elementary School 


(0 Mason-Rice Elementary School 
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Question 6 Findings 


Parents chose academically higher-performing schools if the profiles were sorted by academic performance as 
the default, used icons (i.e., A-F color coded letter grades) to represent performance categories, or included lower 


amounts of information rather than higher amounts.*” 


e Default sort order: Parents could re-sort 
the list of schools by distance, academic 
performance, or school name, but the initial 
(default) sort order mattered.*® When the 
list of schools is initially sorted by academic 
performance instead of distance, parents 
choose schools with higher proficiency rates 
(by about five points, 67 percent versus 
62 percent proficient).*9 


Average academic performance 
of parent's top-ranked schools 
by default sort order 
(Percentage Proficient) 


67.3% 


62.7% 


Difference 
is 4.6% vs 


Sort by Sort by 
distance academics 


Format: The format that added A to F icons 
to the numbers in the display led parents to 
select schools with better academic perfor- 
mance, a difference of more than two points 
between the display with icons and the display 
with only numbers. 


Average academic performance 
of parent's top-ranked schools 
by format 
(Percentage Proficient) 


63.8% 65.0% ls 
Numbers-only Graphs and Icons and 
format numbers numbers 
format format 


Amount of information: The simpler display, 
which had just one measure for each infor- 
mation category, led parents to pick schools 
with better academic performance. This was 
true regardless of whether the profile offered 


Average academic performance 
of parent's top-ranked schools 
by amount of information 
(Percentage Proficient) 


progressive disclosure (had an option to display 66.0% 66.2% 62.9% 

more details). More specifically, the simpler 

display led parents to choose schools whose 

proficiency rate was three points higher on owen Parent chooses Higher 

average than with the higher-information display. information (default is lower) information 
Note: N = 3,500 


Implications 


If encouraging parents to select academically higher performing schools for their children is a goal, evidence suggests that 
the best ways to do so may include initially sorting schools by academic performance, representing academic performance 
with icons that clearly differentiate different categories, and using a simpler display with a lower amount of information. 
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KEY INSIGHTS FROM THE STUDY 


‘The findings for the study’s six questions provide some guidance about information presentation. But looking across 
them suggests three broader insights that information providers can use in designing school information displays. 


1. Decisions about how to display information may require designers to make 
trade-offs between understandability, ease of use, satisfaction, and effects 
on parents’ choices. 

In the study, some design choices forced trade-offs in that they improved information displays for one outcome at 


the expense of another outcome. For example, the study showed that including parents’ ratings of schools and using 
graphs is more satisfying to parents, but both of these strategies make the information display harder to understand. 


Easier to understand, Harder to understand, 
but less satisfying > but more satisfying 


SAFETY SAFETY 


Another example of a trade-off is that it is possible to design information displays to encourage parents to select 
certain types of schools, such as those with higher academic performance, but doing so could unintentionally make 
the information harder to understand and use. For example, sorting schools by academic performance instead of 
distance leads parents to select higher-performing schools, but it also makes the profiles harder to understand and 
less satisfying. Information providers need to decide which outcomes are most important. 


2. All of the different school information displays explored in this guide were 
understandable, easy to use, and satisfying to most parents. 


Large majorities of parents were able to correctly answer factual questions (to demonstrate understanding) and 
agreed or agreed strongly with positive statements about the information displays (indicating ease and satisfaction), 
regardless of which variation they saw.*° For example, while including district averages led to lower ease of use and 
lower satisfaction, parents who saw district averages still agreed or strongly agreed with 86 percent of ease of use 
statements and 90 percent of satisfaction statements, on average.*! 


High rates of understanding, ease of use, and satisfaction suggest that the information display organization used as 
the backbone of this study was itself well designed. Specifically, it is possible to construct the best (and worst) infor- 
mation displays for a given outcome (e.g., understanding) by going through each of the five factors and selecting the 
strategy that improves each outcome the most (or the least). Even the worst set of strategies for information displays 
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had scores for each outcome that might appear reasonable to some information providers. For example, parents who 
saw the information display variation that performed worst in terms of understanding still got 71.1 percent of the 
points correct on the understanding task. 


As discussed earlier in this guide, all of the displays tested in the study used a simple, standard layout with a glossary, a 
map, and four categories of information presented in such a way that it was easy to compare schools. The aspects of the 
displays that were varied were only meant to test potentially promising options against one another. Therefore, it may 
not be surprising that even the “worst” information displays were relatively understandable, easy to use, and satisfying. 


It should be noted that although variations between individual displays were often subtle and none of the varia- 
tions led to what might be considered poor outcomes, information providers have to make several design decisions, 
and the effects can add up. To quantify this, one can identify the combination of strategies—considering all five 
factors together—that produce the best score for a given outcome and compare it to the combination of strategies 
that produce the worst score for that same outcome. This exercise, presented in detail in section C of the technical 
appendix, suggests that there was a difference in parent satisfaction of 5.2 points on the 0-100 scale (percentage of 
items with which parents agree or agree strongly) between the best and worst information display. These design deci- 
sions also had cumulative effects on the academic performance of the schools picked by parents in the study, an effect 
of almost 20 percentile points. That is the equivalent of pushing parents from selecting the 4th ranked school to the 
top-ranked school in the study’s hypothetical school district. 


3. How information is displayed can lead parents to choose different types 
of schools. 


Whether information providers consciously plan to influence choices or not, the ways that information is displayed 
may lead parents to choose schools based on certain categories of information, by making certain types of schools 
stand out. As discussed under question 6, initially sorting schools by academics, including icons, and displaying a 
lower amount of information all encouraged parents to select schools that had stronger academic performance. 


An important insight from the study is that decisions about information displays can influence the schools that 
parents choose in many ways beyond those already discussed. Findings in the technical appendix (see Table C.2) 
show that some strategies tested as part of the study led parents to choose schools that were closer to home or had 
better measures of school safety or resources. Taken together the study findings suggest information providers may 
want to be aware of how specific design decisions may influence the choices parents make. 


People seem to choose based on what is most ; 
Average distance from home 


prominently d isplayed : of parent's top-ranked schools 
. . . ; . coe by default sort order 
One way that information can be prominently displayed is by listing it (Miles) 
first. The study showed that sorting schools by an academic performance 2.3 miles 


measure can lead parents to choose academically higher-performing 1.7 miles 


Difference _/ 


schools, but the same principle can apply to any category of information. EEE 
I : le: 


For example, setting the default sort order to distance led parents to choose 
schools that were more than half a mile closer to home, on average. 


Sort by Sort by 
distance academics 


One possible explanation for these findings is that parents may take 
shortcuts when making decisions. They might look at what is most 
prominently displayed, settle on a choice, and then decide to stop looking further. This phenomenon has been 
demonstrated by researchers in other fields as well.” 
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Another way that information can be prominently displayed is to 

have it take up more space on the page. For example, profiles that 

showed more details about school resources (higher information 
displays) caused parents to be more likely to select schools with 

more resources—5 more laptops per 100 students, on average— 


compared to parents that saw fewer details about school resources 


(lower information displays) (see technical appendix, Table C.2). SAFETY RESOURCES 
It is possible this finding occurred because the profiles with more 91% A2 
details devoted more space to resources compared to other catego- 

ries of information. Specifically, safety and resources each take up 90% ae 


the same amount of space in the profile that shows fewer details 
(“lower amount”), but in the profile that has more details (“higher 
amount”), the description of resources takes up twice as much 
space as the description of safety.” 


Facilities for Extracurricular Activities 


No No Yes 


No No 


Formatting strategies can influence choices by highlighting or obscuring differences. 


Certain formatting strategies can accentuate differences 
between schools while others may obscure or downplay them. 
An example from the study illustrates this point. In the display 


| sarerv® | on the left of the figure (showing icons), the three schools 


Q appear very different in terms of safety, rated as A, B, or C, 
School 1 eaPS wth NO each with a different color. The three schools fall into distinct 
Suspension SuSpencinits 98% categories, each represented by different colors and letters. 
a 
ee ©:: rm However, the display on the right (showing graphs) makes 
students wit Suspensions 95% the three schools look very similar. The horizontal bar graphs 
ens Ee . 
cas are presented on a 0 to 100 scale, which makes the first two 
Students with No : . : 
School 3 .C) Suspensions 88% schools look nearly identical in terms of safety. Therefore, the 


same difference of 3 percentage points (between School 1 and 
School 2) can appear large or small depending entirely on 
visual representation. In fact, the study found that parents who were shown icons for safety measures chose schools 
with better safety scores, compared to parents who were shown graphs of the same data (see Table C.2 in the tech- 
nical appendix). 


A possible reason why icons made safety information more relevant to parents is that the underlying safety scores 

for the schools in the display were closely bunched together. The icon format used letter grades to label numbers that 
would have otherwise appeared similar on a 0-100 scale; this may have accentuated differences in the safety measures 
and led parents to emphasize safety in their school selections. This type of categorization effect has been demon- 
strated in other fields. 


Categorical displays like icons can serve to either accentuate or mask differences, depending on how the categories 
are defined. For example, a district could give an “A” to every school with 80 to 100 percent of its students with no 
suspensions, reserving “B” and “C” ratings for lower percentages. Then all three schools in the example above would 
receive an “A” and parents might be less likely to let safety influence their choice of school. 
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Foemedcuracale Graphical displays can also be designed to accentuate 


Obecures Ditiaroncee or mask differences. For example, the suspension data 


could be presented on an 80 to 100 scale instead of 


School A [IIIS 0, a 0 to 100 scale. The examples below show how the 


zoomed-in scale (80 to 100) makes the schools look 


School B (TTT 95%, different, whereas the otherwise-identical graph with 


a zoomed-out scale (0 to 100) make them look similar, 


School C (I ss. even though the actual differences between the schools 
1 1 are the same in both displays. 
() 20 40 60 80 100 
Students with No Suspensions (%) In short, information providers using graphs or icons 


must choose whether to draw parents’ attention to 
Zoomed-in Scale 


Draws Attention to Differences 


certain differences between schools or to mask those 
differences. Visual formats can be used to accomplish 


‘ either goal. 


School A ( osx 
School B 95% 
School C [NN ss% 


85 90 95 100 


Students with No Suspensions (%) 


LIMITATIONS TO KEEP IN MIND 


This guide is intended to help information providers incorporate evidence from rigorous research into decisions 
about presenting information on schools to parents. However, even the best evidence cannot replace good judgment 
or knowledge of a local context. Nor can it resolve disagreements about which goals to prioritize when a design 
choice is shown to improve one outcome at the expense of another. For example, if a more detailed display improves 
satisfaction but reduces ease of use, this guide does not try to say which is more important. 


The following limitations help to put the research in perspective. 


1. Experimental conditions are the not the same as real-life conditions. 


In a real-world setting, parents can take in official online school information along with other information on repu- 
tation and quality they get through friends and neighbors and direct contact with schools and their staff. In the study, 
the online information display was the only information parents had about schools. Also, the stakes are lower in the 
study because the choices are hypothetical, so parents may be less motivated to review and consider information than 
if they’re making an actual choice to enroll their children in a school. Nevertheless, the study aimed to ensure that 
parents participating in the study took the exercise seriously and the research team removed respondents who tried to 
speed through the survey (a proxy for not responding sincerely). 
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2. The parents used in the study may differ from the parents of interest to 
some readers of the guide. 


‘The study used an “opt-in” approach, meaning that the participants do not statistically represent a well-defined set 
of parents. It is possible that those who sign up to participate in surveys such as the one used in this study, and those 
who put forth effort on the survey, may differ in distinct ways from the national population of low-income parents 
involved in making school choices. For example, those participating in online survey panels may have more expe- 
rience in viewing online information displays than other parents due to their time spent completing surveys. Those 
who persist and attend to the instructions may have a keener interest in the topic. However, because there is no way 
to develop a list containing the national population of low-income parents of school age children who are shopping 
for schools, there is no way to compare the characteristics of the study sample to those in the target population. Also, 
the survey was conducted in English, so it does not represent the potentially large and important group of parents 
who have limited English language skills. 


Despite these limitations, the set of study participants was geographically diverse, representing all 50 states (as shown 
in the technical appendix, Figure B.5), and nearly evenly divided among urban, suburban, and rural respondents (also 
in the technical appendix, Table B.6). 


In addition, most of the findings presented in this guide remain consistent if the sample is restricted to just parents 
with the lowest income, the lowest education, non-intensive Internet use, or with prior experience with school 
choice.* Specifically, of the 18 separate findings presented in the findings and implications section of this guide, 16 
were consistent for parents with the lowest income, 13 were consistent for parents with the lowest education, 17 for 
the parents with fewer than 30 hours of weekly Internet use, and 15 for parents with prior experience with school 
choice. The full set of subsample findings is presented in the appendix, Tables C.3 through C.11. 


3. Some of the findings described in this guide can have more than 
one interpretation. 


‘The guide indicates that parents found the higher-information display more satisfying than the lower-information 
display (Question 4). This could reflect a preference for more detail over less detail, or it could reflect a preference 
for the specific measures that were revealed in the higher information display that were not included in the lower 
information display. For example, the presence of art studios and computer labs was included in the higher but not 
the lower information display. Similarly, the higher information display showed estimated commute time in addition 
to distance. Another example is the addition of parent ratings to the school profiles (Question 3). This led parents to 
find the information display less understandable but more satisfying. This could be caused by the fact that the infor- 
mation was derived from parent surveys or it could simply be another example of the effect of changing the total 
amount of information in the display. 


Despite some of these limitations, information providers should consider the study’s lessons and think about running 
their own studies to test various ways of designing information displays and school profiles if they want to under- 
stand how school information can work best in their areas. Feedback from parents who participate in these studies 
could lead to better, more accessible information and possibly, to better school choice outcomes. Meanwhile, this 
guide demonstrates that information providers already have the potential to design school information displays in 
ways that help parents through the school choice process and ultimately benefit the education system. 
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ENDNOTES 


For simplicity, this guide uses the term “parents” to refer 
to anyone who is making or contributing to the decision to 
choose a school. In reality, other adults—and the students 
themselves (Condliffe, Boyd, and DeLuca, 2015; Valant, 
2014)—often play an important role in choosing a school. 


? Gross, DeArmond, and Denice (2015) describe the 
causes—and potential solutions—to parental confusion in 
the “high-choice” cities of Denver and New Orleans. Stein 
and Nagro (2015) examine one possible source of this confu- 
sion: the excessive complexity of the informational materials 
available to school-choosing families in many large districts. 


3 Haxton and Neild (2012); Stein and Nagro (2015); and 
Stewart and Wolf (2014) have documented the challenges 
that school choice information can pose for families with low 
income, low education levels, or limited English. 


“The National Center for Education Statistics has docu- 
mented the increase in both the number and enrollment of 
charter and magnet schools. See “Table 216.20. Number and 
enrollment of public elementary and secondary schools, by 
school level, type, and charter and magnet status: Selected 
years, 1990-91 through 2014-15” in the Digest of Education 
Statistics, online at https://nces.ed.gov/programs/digest/d16/ 
tables/dt16_216.20.asp last accessed on December 7, 2017. 


°U.S. Congress, “Every Student Succeeds Act of 2015.” Pub. 
Law 114-95 § 114 Stat. 1177 (2015-2016). 


6 Stein and Nagro (2015) 
7See Hibbard et al. (2002) and Hildon, Allwood, and Black 


(2012) for evidence related to presentation of information 
about healthcare plans and providers. 


Section B.5 of “Every Student Succeeds Act State and 
Local Report Cards Non-Regulatory Guidance” January, 
2017. Accessed at https://www2.ed.gov/policy/elsec/leg/essa/ 
essastatereportcard.pdf, December 7, 2017. 


* The study was conducted by Mathematica Policy Research. 


10 Harris and Larsen (2015); Haxton and Neild (2012); Stein 
and Nagro (2015); Stewart and Wolf (2014) 


"This part of the study also introduced a special survey item 
called an instructional manipulation check (IMC). The IMC, 
which is described in more detail in the technical appendix, is a 
question that is easy to answer correctly if one reads the instruc- 
tions, but tempting to answer incorrectly if one does not. This is 
designed to check whether participants were complying with the 
study’s instructions and to encourage them to do so. 


” The majority of parents in the study (70 percent) reported 
that their youngest child was elementary school aged or 
younger and, in turn, the hypothetical scenario was likely easy 
for these parents to imagine. 


3 By assigning parents at random to one of the displays, the 
researchers could be sure that the only difference, on average, 
between groups of parents is the information display that 
they saw. This makes it possible to conclude that any differ- 
ences in outcomes were caused by the information display 
strategy being studied. 


“This is not a comprehensive list of the categories of 
information that parents consider. Notably, the school 
profiles omitted student demographic data although research 
indicates that parents consider student demographics when 
choosing schools. Including demographic information would 
have complicated interpretation of the study’s results. 


15 Glazerman and Dotter (2017); Harris and Larsen (2015); 
Hastings, Kane, and Staiger (2009); Schneider, Teske, and 
Marschall (2002); Stewart and Wolf (2014) 


16 Many of these principles are drawn from research on web 
design. See Hoekman (2010) 


™ Schneider and Buckley (2002); “Every Student Succeeds 
Act of 2015.” Pub. Law 114-95 § 114 Stat. 1177 (2015- 
2016). 


18The study used Bayesian inference, described in the 
technical appendix, to calculate the probability that any 
given difference was greater than zero. As discussed in the 
appendix, the study only reports on results where that poste- 
rior probability was at least 70 percent. 


With five factors, each with two or three strategies, there 
are 72 possible combinations of strategies. Formally, the study 
design is a 3x2x2x3x2 factorial experiment. The technical 
appendix contains details about the study’s methodology. 


?° Harris and Larsen (2015); Dynarski et al. (2016); 
Glazerman and Dotter (2017) 


21 See Hibbard et al. (2002), Borgmeier and Westenhoefer 
(2009), Jones and Richardson (2007), and Jacobsen et al. 
(2014). 


2 Glazerman (2017) describes and provides links to 14 
Internet sites that provide such information. Two exam- 
ples are the My School DC school finder at http://find. 
myschooldc.org/ and Great Schools Detroit at https://www. 
greatschools.org/michigan/detroit/schools/, both accessed 
September 20, 2017. 


*5'The current study retained the numbers in all cases to 
ensure that all parents in the study had at least an exposure to 
the common information which was used in testing under- 
standing. 


*4For each measure where letter grades were shown, grades 
were defined based on the relative ranks of the 16 schools in 
the display: ranks 1-4 received an “A,” ranks 5-8 received a 
“B,” ranks 9-15 received a “C,” and the school ranked 16th 
received an “F.” Parents did not receive an explanation of how 
the grades were defined; however, the value of the relevant 
measure was shown as a number next to each letter-grade 
icon, so it was possible to identify the cutoff value between 
grades by reading the information in the display. 


5 Kahneman (1992); Kahneman and Tversky (1979); Thaler 
(1985) 


26 Schneider et al. (1998); Schneider, Teske, and Marschall 
(2002); Bell (2009); Valant (2014) 


27Valant (2014) 
8 Cronqvist and Thaler (2004); Peters et al. (2007) 
?? Bundorf and Szrek (2010); Chakraborty et al. (1994) 
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%°'These pieces of information include “the number and 
percentage of students at each of three or more levels of 
achievement on each of the academic assessments in mathe- 
matics, reading/language arts, and science”, the number and 
percentage of English learners achieving English language 
proficiency, four-year adjusted cohort graduation rate, 

and whether and for what reason the school is identified 

for comprehensive or targeted support and improvement. 
Furthermore, this information must be presented separately 
for economically disadvantaged students, students from each 
major racial and ethnic group, children with disabilities, and 
English learners. See “Every Student Succeeds Act of 2015.” 
Pub. Law 114-95 § 114 Stat. 1177 (2015-2016). 


31 Glazerman (2017) 
52 Glazerman (2017) 


53 Sorting by name (alphabetical sort) is also common, 
although this strategy was not included in the study because it 
assumes that parents are already familiar with the schools and 
would want to look up specific schools about which they had 
prior knowledge. 


34 Hibbard et al. (2002); Hildon, Allwood, and Black (2012) 
35 March (1994); Simon (1955) 


36 Akaah and Korngaonkar (1983); Jansen et al. (2000); 
Lewinsohn and Mano (1993); Srinivasan and Park (1997); 
Thaler and Sunstein (2009) 


3” Every factor that was tested affected the types of schools 
parents chose. This section highlights particular strategies 
that encouraged parents to choose academically higher-per- 
forming schools. The full set of findings summarizing how 
display strategies influenced the types of schools selected 
by parents is shown in the technical appendix, Section C.1, 
Table C.2, and discussed later in this guide. 


8Tt “mattered” if the probability that there was a positive 
effect was at least 70 percent. Only findings that met this 
threshold are discussed in this guide. The probability is calcu- 
lated as from the posterior distribution of a Bayesian model 
described in the technical appendix Section B.1. 


°° The vertical axes on all of the figures goes from 45 percent 
to 95 percent proficiency, which represents the lowest and 
highest proficiency rates of the schools used in the study 
(excluding one school that was created as an outlier on 
purpose to test readers’ attentiveness). 


“Descriptive statistics for these outcomes are presented in 
the technical appendix, Tables B.1 and B.3. 


“This guide refers to predicted averages, which are based on 
the whole sample, not just parents assigned to one particular 
combination of factors. 


” Browne et al. (2007) 


“8 Another explanation could be that parents found the new 
information (such as whether there is a music room, art 


studio, and so on) to be especially appealing. 
44 McKelvie (1978) 


* The sample included parents in urban, suburban and rural 
areas. While distance or commute times to school could be 
interpreted differently in urban areas than in suburban or 
rural areas, the experiment had an insufficient number of 
respondents from urban areas to allow for separate analyses 
by urbanicity. Among parents in the experiment, living in 
an urban area was associated with more exposure to school 
choice and with being lower income (chi-square tests for 
independence are rejected at the 0.001 level for both asso- 
ciations, as shown in the technical appendix B). These two 
characteristics, prior choice exposure and being low income, 
are two aspects of urban areas that may affect how individ- 
uals experience school choice information, and the study 
examined these two groups of parents as subsamples. 
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